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ABSTRACT 

The objective of this study was to investigate a pro- 
posed method of auditing cost-effectiveness analyses of 
technological changes. The method is based on the use of cost 
functions to predict costs of operating with and without the 
improved technology. A comparison of these predicted costs 
indicates whether the technological change is cost-effective. 
Results of the comparison can then be used to audit more 
detailed analyses of the change. The investigation employed 
econometric techniques in a case study application of the 
method to a technological change which occurred at the Naval 
Air Rework Facility, North Island, San Diego, California. 
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INTRODUCTION 



I . 

A. DEFINITION OF THE PROBLEM 

It is commonly accepted that technological advancements 
are arriving at a faster and faster pace with the state of 
the art generally far ahead of the functional application. 

One reason for this gap is the costly enterprise of transfer- 
ing theory into practice. When the initial transfer is made 
a natural question to ask prior to extensive implementation 
is "Will it be cost-effective?" Current methods of cost- 
effectiveness analysis are tedious and usually employ a 
detailed analysis of the process which incorporated the 
change. Such analyses are inherently imprecise due to the 
possibilities of overlooked dependencies, double counting 
of costs, and external changes which may affect the effective- 
ness of the process and yet not be accounted for in the 
analysis. Thus when a cost-effectiveness analysis is com- 
pleted a reviewing agency might well ask, "Are the conclusions 
of the analysis correct?" As mentioned, the methods em- 
ployed in the analysis make this second question a hard one 
to answer . 

This study is solely concerned with investigating a 
method which may simplify the task of answering the second 
question; that is, a method for auditing cost-effectiveness 
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analyses of technological changes. The investigation was 



accomplished by applying the method to a specific technolo- 
gical change which occurred at the Naval Air Rework Facility 
at North Island, California (NARFNI ) . While an indirect 
product of this study is a statement concerning the cost- 
effectiveness of the change which occurred at NARFNI, it is 
emphasized that the purpose of the study is to investigate 
and refine an auditing method initially proposed by Robert 
,L. Spooner ^^ef . 4^, which was based on the estimation of 
production functions through linear least-squares regression 
techniques. A description of Spooner's proposal and later 
refinements established by Charles W. Bradley and 

Wilber C. Trafton ^ef . 6^ will be given in Chapter II. 

B . NARFNI 

NARFNI is one of seven naval air rework facilities 
throughout the continental United States responsible for 
major maintenance , conversion and repair of United States 
Navy and Marine Corps aircraft and related components . To 
accomplish this mission NARFNI employes approximately 6800 
civilian workers and spends $150 million on annual operating 
expenses ^^radley, pp. 8 and 12/ . 

Maintenance of equipment is accomplished under one of 
three primary programs, with two of the programs receiving 
inputs either directly from the customer or from the third 
program (see Fig. 1). 
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FIGURE 1. NARFNI Repair Programs 

Under these programs work is further assigned to one of 
the six divisions of the Production Department identified in 
Figure 2 . 



V^/eapons #1 
Weapons #2 
Hydro lic/Mechanical 
Power Plant 
Avionics 

Components and Metal 

FIGURE 2. Production Department Divisions 

The assignment of work to various programs and production 
divisions can be thought of as a "job shop"' operation in 
which! each shop performs a specific task. Thus an item 
entering rework is subject to being dismantled, its sub- 
assemblies distributed for rework, and then reassembled prior 
to returning to the customer , This type of operation places 
emphasis on the scheduling of rework in that ideally the 
shops are completing work on related parts in the same se- 
quence to reduce the total repair time of the major end items. 
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Scheduling is also important to the shop managers who must 
order material and plan personnel assignments based on the 
projected arrival of work from other shops. This entire 
scheduling problem is intimately related to the technological 
change to be described in the following section. 

NARFNI is primarily funded by the Navy Industrial Fund 
(NIF) which provides a working capital fund to finance repair 
operations . The finished product is returned with a bill 
to the customer for the work done. This "debt" is then paid 
by transfering funds from the customer's appropriated main- 
tenance funds to the NARF ' s working capital fund. Under this 
concept NARFNI is required to control its finances to incur 
zero profit at the end of each fiscal year. This control is 
exercised at Quarterly Planning Conferences in which repre- 
sentatives from the customers and NARFNI meet to plan the 
following quarter's work load input and prices for the work 
to be accomplished. 

The work load is measured in number of required man 
hours (NORM) of direct labor needed to accomplish the work 
requested by the customers. The NORM is estimated from 
historical data maintained by NARFNI on each type of work 
it has accomplished in the past . 

NARFNI is reimbursed under basically tvro types of con- 
tracts, cost reimbursable and fixed price, the majority of 
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the work being performed under fixed price contracts estab- 
lished at the Quarterly Planning Conference. This type of 
contract gives NARFNI incentive to minimize costs subject 
to the required work load, a fact which is instrumental to 
the auditing method investigated. 



C. WORK IN PROCESS INVENTORY CONTROL SYSTEM (WIPICS) 

In January, 1972 , NARFNI installed an industrial informa- 
tion system to assist in the location and scheduling of 
inventory in the rework process . WIPICS was requested and 
initiated by the Management Systems Development Office (MSDO) 
in an effort to assist the NARF ' s in performing their mission 
at a minimum cost to the government . If the prototype at 
NARFNI is determined to be cost-effective WIPICS will then 
be installed at the remaining six NARF ' s . Its major milestone 
are listed in Figure 3. 



1. Early 1969 - ROHR Corporation studied NARFNI 

2. Late 1969 - MSDO requested contract 

3. Early 1970 - NAVAIRSYSCOM evaluation and approval 

4. July 1970 - SECNAV authority for prototype 

5. Late 1970 - Contract negotiated 

6. 4 January 1971 - Contract date, D-Day 

7. D + 9.5 months - WIPICS developed 

8. D + 11 months - WIPICS test 

9. D + 11.5 months - WIPICS prototype started 

FIGURE 3. WIPICS Milestones 
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WIPICS consists of a central computer which is linked 
to the job shops by 164 "touch tone"' telephones, 20 alpha- 
numeric terminals, and four teletypewriters. In addition, 
the system is equipped with an 80 word audio-response unit 
to allow two-way communication through the telephones. When 
an item first arrives at NARFNI it is broken down into 
identifiable subassemblies. For each of these subassemblies 
a computer record is created and entered into WIPICS. This 
record contains information such as identification, required 
work, required material, location, and status. Each job 
shop inputs arrival and departure information through the 
"touch-tone"' telephones. Also, an unexpected delay in work 
due to lack of material or some other shortcoming is inputed 
as it occurs. At the completion of the work day a print-out 
of this information is produced to aid shop managers in 
projecting the rate and mix of work that will be entering 
their respective shops in the near future. 

The inauguration of the WIPICS prototype was accomplished 
by creating computer records for all jobs presently in the 
shop to preclude a lengthy start up before evaluation could 
begin. Simultaneously, the operational costs accountable to 
WIPICS were transfered as overhead to be applied to the price 
of jobs in shop. Thus, during January, 1972, NARFNI initiated 
a technological change which was to be subjected to 
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cost-effectiveness analysis and, at the same time, provided 
a suitable situation to use in investigating a method for 
auditing cost-effectiveness analyses . 



D. SCOPE OF STUDY 

Previous work conducted by Spooner, Bradley, and Trafton 
was directed at completing a methodology for determining 
if NARFNI is operating more efficiently after installation 
of the WIPICS prototype. However, the method is relatively 
new itself and numerous problems prevented completion of 
the effort. One major problem was the lack of production 
data after the incorporation of WIPICS. The purpose of this 
study is to complete that work. The remaining portions of 
the. study are divided into the following catagories: 

1. Discussion of the methodology and production data 
available from NARFNI . 

2. Preliminary estimation of NARF costs as functions 
of the work load and prices of inputs, and analysis 
of the characteristics of these functions. 

3. Estimation and analysis of before and after WIPICS 
cost functions followed by a direct comparison of the 
two cost situations. 

4. Discussion of the exogeneous variables which were 

’ not included in the estimation of the cost functions. 

5. Conclusions. 



13 



II. 



METHODOLOGY AND DATA 



A . GENERAL 

The instructional objective is to determine the effective- 
ness of WIPICS, The major difficulty in attaining this 
objective is the probable abundance of other changes which 
make the before and after WIPICS situations different. Thus, 
a direct comparison of costs is invalid. The ideal solution 
would be to operate two ’'identical” NARF ’ s , one with and one 
without WIPICS, and then compare costs., While this solution 
is unobtainable an estimation of its outcome is not. The 
costs of those two NARFs can be estimated through the use of 
econometric techniques to construct before and after WIPICS 
cost functions. The estimation methods were reported by 
Trafton but are outlined in Sections B, C, and- D for reference. 

In order to estimate the cost functions their structure 
must., first be specified by solving a mathematical program in 
which NARFNI is assumed to be minimizing costs subject to a 
production function, 

minimize C = P..X t . . . Pn^r> 

linn 

subject to Y = F(X]^,..X^). 

As stated in Chapter i , NARFNI has three major programs, 
each producing many different outputs. For example, the 
airframe program produces airframe repairs on 20 quite 
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different aircraft, and the engine program has an even more 
varied output. To obtain a more tractable situation the 
problem at (1) was solved and then applied to the major 
NARFNI programs separately. The result is one cost function 
for the airframe program and one for the engine program. 

Before describing the form of the cost functions a description 
of the data available and the types of production functions 
assumed to hold is necessary. 

B. NARFNI PRODUCTION DATA 

The data was obtained from NARFNI ' s Production Performance 
Report, a quarterly report which details the total resources 
expended on each airframe and engine which has completed its 
repair process. The report does not include detailed infor- 
mation on the component parts program but rather a monthly 
summary of total resources expended in that program. A 
description of the data for airframes and engines is given 
below . 

1. Identification - includes the type of aircraft or 
engine and the extent of work required, such as 
overhaul, repair of crash damage, modification, or 
conversion. The type of work required affects tlie 
NORM assigned to the item . 

2. NORM - is NARFNI ' s estimate of the direct labor hours 
required to accomplish the work. NORM is established 
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at the Quarterly Planning Conference before in- 
duction of the item. 

3. Induction and completion date - are the Julian dates 
the item actually entered rework and then work 
completed . 

4. Direct labor hours (L) - is the total labor hours 

that can be specifically identified with rework of 
the item. 

5. Direct labor dollars (L$) - is the total cost of 
the direct labor expended on the item. 

6. Material dollars (M$) - is the total cost of materials 
that can be specifically identified with rework of 
the item. 

7. Overhead dollars (0$) - is a prorated portion of 
those NARFNI costs which cannot be identified with 
specific items in rework. 

The data taken from the Production Performance Reports 
are raw in the sense that they represent actual costs per 
item. Bradley wrote a computer program which prorates the 
NORM, DLH, DL$ , M, O for each item over the number of days 
the item was in shop. The program then aggregates the pro- 
rated data to obtain daily totals for the engine and airfrajne 
prograjms for the period under study. Bradley's program 
generates one additional variable, the number of items in 
rework (I) on each day. Trafton modified these programs 
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to include a penalty cost (P$) chargeable to each item for 
every day it was in the NARF p. 22^. This was an effort 

to reflect pipeline costs which may be the costs most sig- 
nificantly altered by WIPICS . These penalty costs are 
especially significant in the component parts program. A 
relatively small part, such as' a generator, may be the prime 
reason an F-4 aircraft is non-operational at its home base. 
Unfortunately, lack of data in the component parts program 
prevented inclusion of that program in this thesis. 

Two variables were constructed from the aggregated data. 

L$ 

PL = which is a rough surrogate for the price 

L of labor , and 

PI = P$ , which is an average penalty cost per item 
b on a given day . 

Notice that PL will vary with changes in the amount of 
overtime employed as well as a general pay raise. It may 
also vary as the distribution of jobs in the work load 
changes from one level of skill to another. 

C. PRODUCTION FUNCTIONS 

Two structures for the production function in the economic 
problem of Section A were used, the Cobb-Douglas and the 
Constant Elasticity of Substitution (CES) production func- 
tions. This was done to compare the ability of each to 
model the relationship between NARFNI ' s inputs and outputs. 

In each program the output is approximated by NORM produced 
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each day. This approximation is necessary due to the numerous 
types of airframes and engines being repaired. Since NORM 
is measured in the same units for every item it is a simple 



way of measuring production while at the sarae time reflecting 
the amount of work accomplished by NARFNI each day . 

The Cobb-Douglas production function was assumed to have 
three inputs: 

1. L with price PL 

2. I with price PI 

3. M with price 1 (since material is m.easured in dollars). 
Thus the production function takes on the form 



The CES production function was assumed to have two inputs, 
L and I , and takes the form 



The Cobb-Douglas production function was selected for its 
useful and convenient application in emperial work. It is 
a special case of the CES function in that the limit of the 
CES production function, as ^ o , is the Cobb-Douglas function. 
Therefore, the elasticity of substitution (<r) of inputs is 
assumed fixed at one in the Cobb-Douglas function and it is 
only assumed to be a constant in the CES function ^2, p. 32.1J . 



InN = cXq InL -f ^2 Inl -f InM . 



( 2 ) 




(3) 
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The CES producti.on function was included in the study 
specifically because it allows to differ from unity and 
therefore, should present a more accurate model of NARFNI ' s 
production relation should this be the case. This advantage 
is offset, however, by the fact that the CES function at (3) 
above is not linear in the parameters and estimation of the 
resulting cost function becomes more difficult. 

D. ESTIMATING THE COST FUNCTIONS 

The following procedures apply to both the airframe and 
engine cost function estimation . For this reason only the 
airframe procedures are presented. 

1 . Cobb-Douglas Cost Functio n 

The cost function is derived by solving 
min C = PL(L) + M + PI(I) 

, Q ^ 

St. N = AL I M /see 6, p. s/7 

where the dependent variable in the objective function is 
defined as DL$ + M + P. The Lagrangian (LG) is 

LG = PL(L) + M + PI(I) - X(N-AI? I^M^) . (4) 

The 1st order conditions are obtained by taking partial 
derivatives of (4) with respect to L, M, I, and . The results 
are : 



-1 ^ r 

LG^ = PL +AcCAL r M 


(5) 


UG^ = 1 + ATAL^I^M 


(6) 


LG^ = PI + A/3Alf 


(7) 
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cK. 6 ^ 
N-AL 



( 8 ) 



LG^ = 

where the 's are random variables associated with errors 
due to imperfect cost minimization by NARFNI . 

These 1st order conditions can be solved to obtain 
the inputs as functions of the prices and output . Substitu- 
tion of these functions into the logarithmic form of the 
cost function yields 

InC = a + ^ + y InN + InPL + InPI + £ (9) 

where the error term (€) is assumed to be normally distri- 
buted with mean zero and constant variance. 

Standard linear least -squares regression techniques 
/p , Chap. ^ were used to estimate the coefficients of the 
variables in (9) . The results are reported in Chapters III 
and IV. 

2 . Constant Elasticity of Substitution Cost Function 
Again, the cost function is obtained by solving 
min C = PI(I) + PL(L) 

St N = r|^L"^ + (l-<f) 

where C is now defined as DL$ + O + P. The resulting function 
is InC =oC+ In |pL + PI(^/l)J + ^ InN +«^D + £ (10) 

in which 



oC = 



- i In JT 




D = In 



( 1 -^) + 






( 11 ) 
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The equation at (10) is not linear in the original 
parameters of the production function and, therefore, cannot 
be estimated by the same procedures used on the Cobb-Douglas 
cost function. A more complex two-stage least -squares re- 
gj- 05 sion technique was used instead. The first stage consists 
of a linear least-squares estimation of 

In (^/L) = a + b ln(^^/PL) + 6 (12) 

where 



— . n . 

a = (ilL^) ^ and b = -P-1 

6 

Equation (12) is derived from the first order conditions of 
the mathematical program /see Trafton, p. 18^ . The result- 
ing estimate of (^/L) is then substituted into D at (11). 

The estimate of D is then substituted into (10) which becomes 
linear in the parameters. Thus, the second stage consists 
of a linear least-squares regression to estimate the co- 
efficients of (10). 

A comparison of the Cobb-Douglas and CES cost functions 
derived is discussed in Chapter III where a preliminary 
analysis of the procedure is reported prior to application 
of the entire auditing methodology . 



21 



E. DIRECT COMPARISON OF THE ODST FUNCTIONS 



To estimate the effect of WIPICS on NARFNI ' s efficiency 
the cost functions derived in Section D were estimated using 
aggregated production data from a period of time prior to the 
inauguration of WIPICS to obtain before-WIPICS engine and 
airfrcime cost functions. The procedure was then repeated 
using aggregated production data for a period after WIPICS 
for an after-WIPICS cost function for each program (Fig. 4). 





Before WIPICS 


After WIPICS 


Airframes 


CD cost function 


CD cost function 




CES cost function 


CES cost function 

! 


Engines 


CD cost function 


1 

CD cost function 




CES cost function 


CES cost function 



FIGURE 4. Cost Functions Estimated 

To compare the two situations the before and after data 
were consolidated and used in the before and after cost func- 
tions to estimate daily program costs for a given period. By 
averaging the difference between the expected cost with 
WIPICS and the expected cost without WIPICS a mean cost 
differential is predicted for each of the following situations 

1. NARFNI operation with and without ' WIPICS using before 
WIPICS observations. 

2. NARF?'!! operation with and without WIPICS using after 
WIPICS observations . 
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This procedure is followed for both the Cobb-Douglas and CES 
functions. The computer program constructed for this purpose 
is listed at Appendix A. 

F . DATA 

Production data was obtained on the engine program for 
two separate periods. The first period covered 1865 engines 
inducted on Julian dates 0140 through 1273. The aggregated 
results obtained from this period is listed in Appendix B 
of Bradley's thesis. The second period covered 521 engines 
inducted on 2003 through 3050 with the aggregated results 
listed at Appendix B of this thesis . Data on the airframe 
program covered 564 airframes inducted on 0069 through 2234. 
The aggregated results from Julian date 1001 forward arc 
listed in Appendix C. 

The operational cost of WIPICS chargeable to NARFNI 
averages approximately $48,000 per month or $1600 per day. 
NARFNI was not held accountable for these charges until 
January, 1973. 
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Ill . 



PRELIMINARY ANALYSIS 



Before attempting to apply the procedures outlined in 
Chapter II a preliminary analysis was performed to develop 
methods of characterizing the estimation techniques. This 
was accomplished by using a 400 day period of data to esti- 
mate cost functions and test for stability and accuracy. 

The resulting proceedures were then included in the final 
analysis of the technological change. The 400 day period 
used in this chapter was well before the WIPICS prototype 
became operational. Therefore, the results obtained should 
reveal some of the problems to look for in the estimation 
of the cost functions, that is, the expected variability of 
the cost function due to events other than the implementation 
of WIPICS. 

A . REGRESSION RESULTS 

Using the procedures outlined in Chapter II Cobb-Douglas 
and CES cost functions were estimated from 300 days of 
aggregated production data. The resulting coefficients are 
presented in Table I. The results of the first stage of the 
estimation procedure for the CES cost function is not pre- 
sented unless the coefficients are not significantly different 
from zero. R^ , when multiplied by 100, is the percentage of 
total variation in cost that can be explained by the estimated 
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function. SE is the standard error of the estimated cost 
function (in logarithmic form), and the number in parenthesis 
is the standard error of the coefficient just above it. 
Standard errors for leading coefficients are not computed. 



Airframes 



1 



Engines 



2 



— 

! 

( Function Coefficients 


1 

SE i 


1 


j CD: .624 .760 2.754 -.267 

1 (-015) (.066) (.036) 

i 


, .021 


i 

.956 : 


CES: .2175 0.870 -.2.461 

(.007) (.182) 


.011 


.980 


! CD; 3.046 .877 .401 .032 


1 .035 


.907 


1 (.017) (.095) (.052) 

j 


\ 

1 

\ 

\ 




j 

icES: 2.506 .851 21.152 

1 ( .025) (6 . 113) 

i 


1 

1 .053 

1 

X 

i 

! 


.808 



TABLE I. Before WIPICS Cost Functions (Preliminary) 



300 observations from 0191 to 1226.' 
^ 300 observations from 0215 to 1150 
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It should be noted that the dependent variable of the 



CES second stage regression is not InC but rather 



In 



C 

PL + PI(-*/L) 



Also, the C representing costs in the Cobb-Douglas function 
is the sum of material, penalty, and labor dollars, where as 
the C in the CES regression is overhead, penalty, and labor 
dollars. These two differences may explain why, in the air- 
frame program, the CES function has a higher R than the 
Cobb-Douglas (indicating a better fit) while the Cobb-Douglas 
has a lower standard error (indicating it is more accurate). 
The two functions are thus not comparable in this form, so 
different methods of comparison were needed. 

A rough test for the significance of the coefficeints 
in Table I can be made by taking the ration of. the coefficient 
to its standard error . If this ratio is less than two the 
coefficient is not significantly different from zero at 
approximately the 5% level of significance. All coefficients 
in Table I are significantly different from zero by this 
test and the conclusion can be made that each independent 
variable contributes to the explanation of the dependent 
variable . 
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B. STABILITY OF THE MODELS (CHOW TEST) 



With the large fluxuation in work load and type at NARFNI 
(see NORM and penalty cost values in Appendices B and C) it 
is probable that the production and cost functions for NARFNI ' s 
programs shift considerably even though technology is in 
fact not changing. Inherent instability such as this may 
camouflage the shift due to technology and should, therefore, 
be tested for by means of the Chow Test , pp . 104-1127. 

The Chow Test is founded on the general principle that a re- 
gression line will not pass through every data point and that 
data obtained from two different populations can be better 
fit with a regression for each population than with a single 
regression for the total data set. If the converse of the 
above statement is assumed to hold, then a test to see if two 
sets of data are from the same population may be obtained by 
seeing if a single regression fits as well as two separate 
regressions on the respective data sets. To extend this test 
to the cost functions a second regression was performed on 
the 100 day periods following the 300 days in each of the 
two programs under study. The data was then combined and 
regressions run on the 400 day periods. The test statistic 
used was 



F(k ,n 4-2k ) 



^ 4 (SE^) - n^ ( SEg ) -n j ( SE^^ ) n^-2k 

n3(SE3) + n^(SE^) 2k 
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The subscripts refer to the respective data periods of 400, 
300, and 100 days. SE is the resulting equation's standard 
error, n is the number of observations, and K is the number 
of coefficients estimated in each equation. If the test 
statistic is too large then the hypothesis that the two 
periods have the same cost function is rejected. Results are 
presented in Table II . 

Test 





Model 


Statistic 


Critical Value Results 


Airframes 


CD 


12.147 


F(4,392) 

2.43 


i Reject 1 




CES 


28.78 


F(5,390) 

2.27 


1 Reject i 


Engines 


CD 


14.741 


F(4,392) 

2.43 


I Reject 1 




CES 


41.22 


F(5,390) j 

2.27 1 


1 

Reject 1 



TABLE II. Chow Test Results (Preliminary) 

The entire 400 day periods used in the Chow Test fell 
well before WIPICS was operational at NARFNI indicating the 
cost functions are not stable, even with fixed technology. 
This revealed a possible problem in applying the auditing 
methodology proposed by Spooner . However , it may be possible 
to dimenish the importance of the instability if it is not 
too great . This can be done by increasing the sample size to 
well over 400 days, thereby allowing the variations in work 
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load to average out . This technique would prove ineffective 
against large, prolonged projects of a particular work type 
such as major modification of all F-4 aircraft in the Navy’s 
inventory. In addition, this solution would mean waiting, 
perhaps for years, before obtaining enough data, after the 
technological change, to conduct the analysis. In most cases 
the requirement for the study would have long since been 
dropped . 

A more generally acceptable solution would be to accept 
the cost function for use as long as it can accurately pre- 
dict costs while at the same time keeping the accurracy in 
mind when final conclusions are drawn. This acceptance can 
only be justified if the accuracies of the function are 
measured. 

C. ACCURACY OF THE COST FUNCTIONS 

1 . Empirical Confidence Interval s 

t A 

If Sj is the error of the predicted cost for the i 
observation a confidence interval can be empiriceify constructed 
by using the ratio of two standard errors^ of the function 
over the actual mean costs for the period. The result is a 
percent error in cost, 

3 

The term "standard error" is miss-used here since 
nothing is known about the distribution of the error terms. 
Still, the procedure is not uncommon. 
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% error 



X 100. 



2 'I 

The % error approximates + 2 standard deviations which 
accounts for about 90% of the mass in most probability 
distributions. Thus, there is approximately a 90% confidence 
that the predicted cost is within the percent error com- 
puted of the actual cost . 

This procedure was performed on the cost functions esti- 
mated from the 300 day period, and then on the 100 day period 
with costs being predicted by the 300 day models (Table III). 

Model 300 Observaticns 100 Observations 

Airframes 
Engines 



CD i 


1 04.0% 


± 05.0% 


CES i 


1 02.3% 


± 03.1% 


CD 


± 07.0% 


1 15.7% 


CES 


± 11.1% 


1 . 14.6% 



TABLE III. % Errors in Cost Functions (Preliminary) 

As the standard errors seemed to indicate in Table I, 
the CES cost function appears better for the airframe pro- 
gram while the results are mixed for the engine program. 

The largest errors are in , the engine program when the functions 
were used to predict costs. This is understandable since 
the engine work load and mix of work fluctuate more rapidly 
and extremely than those of the airframe program. The re- 
sults also indicate that the engine cost functions may be 
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highly sensitive to small shifts in the data period used. 
Finally, in the 300 day functions for engines, the Cobb- 
Douglas model appears to do a better job than the CES . Thxs 
is counter to expected results and is not explained. 

2 . Statistical Prediction Intervals 

A similar procedure to that of subsection 1 uses 
statistical methods outlined in Theil ^p • 134-13^ to con- 

struct prediction intervals. Let c' represent the estimated 
cost and accept the assumptions of the standard linear model 
pp . 110-113^7. Then the following probability statement 

holds: 

P(lnC| - H ^ InC^ ^ InC^ + H) = 1-di 
where 

H= t%(SE) y 1 + v; (x'x)-^Vo 

X = matrix of independent variables used to construct 
the model 

Vo = vector of independent variables used to predict 
InCi 

"t" -statistic 

/x 

This interval is in logarithms. The transformed interval 
in costs becomes non-symetric and non-linear in the xnde- 
pendent variable of Vo. Thus, it is more accurate to compare 
the extreme widths of the intervals than the mean values 
compared in subsection 1 . 
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Prediction intervals were computed for the 100 day 
period using the 300 day models of Table I . Of the 100 
prediction intervals resulting the minimum and maximum in- 
terval were transformed to dollars and then divided by the 
mean cost for the 100 day period to obtain a maximum and 
minimum width as a percentage of the average cost (parenthe- 
tical entries of Table IV). Finally, the percentage of actual 
costs that fell outside their corresponding prediction 
interval is reported in the last column of Table IV. Com- 
puter programs written to perform these computations are 

reproduced at Appendices D and E . 

Max. Min. % Out- 

Model Interval Interval side Interval 

Airframes 



Engines 



CD 

CES 


6729 

(8.5%) 


5365 

(6.8%) 


4% 


4480 

(4.0) 


3347 

. 3^0 ) 


32% 




5375 


3783 


62% 


' CD 




(8.8) 






6943 


4836 ■ 


29% 


CES 


(20.5) 


(14.3> 





TABLE IV. Extreme Widths of 90% Prediction 
Intervals (in dollars) 



3 . Accuracy of the Measures 

Table III and IV should be roughly comparable by 
doubling the % error under the 300 observations column of 
Table III. This value should fall between the maximum and 
minimum percentage values for the same function in Table IV. 
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As can be seen, the statistical values of Table IV tend to be 
smaller when this comparison is made. Moreover, a larger 
discrepancy between the two tables is accompanied by a larger 
percentage of the actual costs falling outside of the pre- 
diction interval of Table IV. 

Some of this difference may be accounted for as 
interpolation errors for the t -statistic used in the equa- 
tion at (7) to obtain Table IV. Errors in this statistic are 
exponential in nature and the resulting prediction interv'als 
were found to be quite sensitive to the value used. The Scime 
-statistic value was used for all of the functions in . 
Table IV which seems to imply the value chosen was low. A 
slight increase in this value would increase the prediction 
intervals, make Table IV more consistent with Table III, 
and should, at the same time, decrease the percentage of 
actual costs falling outside the corresponding prediction 
interval . More care was taken in selecting this value in 
Chapter IV where the auditing methodology is carried through 
completely . 

Finally, the uneven prediction results between the 
cost functions of Table IV may be indicative of the Chow 
Test's conclusion that the two periods of data, 300 vs. the 
following 100 days, are in fact different and a model of one 
period is a poor prediction of costs in another period. This 
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possibility is important because the periods are not separated 
by a recognizable technological change. It reemphasizes the 
need to include the accuracy of the models in the final com- 
parison of estimated costs. 
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rv. COMPARISON OF BEFORE AND AFTER 



WIP ICS PREDICTED COSTS 



A. THE COST FUNCTIONS 

The final cost functions used to compare estimated costs 
across the technological change were obtained using the 
SNAP/lEDA Computing Package developed by the Department of 
Statistics, Princeton University. SNAP/lEDA will accept 
up to 400 observations. In those models requiring over 400 
observations the BIMED-02R prograjn, developed by the Health 
Sciences Computing Facility, University of California, was 
used for the Cobb-Douglas estimation. The CES estimation 
was limited to 400 observations in all cases since estima- 
tion using the BIMED-02R program would become a 3 stage 
procedure requiring excessive manipulation and ^ inefficient 
use of computer time. Also, for the final cost functions, 
the data periods were altered for two reasons. First, the 
data periods were chosen close to the technological change 
so the before and after situations would be as similar as 
possible, except in technology. Second, the data periods 
were altered in size and content to reduce the effects of 
instability as much as pos.sible . The results are presented 
in Tables V and VI with footnotes indicating the data used. 
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Note that the CES function for engines across WIPICS 
was not estimated due to the large number of observations 
involved. Omission of this function merely precludes per- 
forming a Chow test which in all probability would be rejected 
if performed. Visual comparison of the before and after 
functions supports this. 





Function Coefficients 






SE 


2 

R 




j 

CD 


i 

[- .297 


.857 


2.356 


-.094 


I 

.014 ! 


.949 


Before . 




1 


(.011) 


( .090) 


(.015) 


j 

i 




WIPICS 


CES , 


t 

1 

[2 .999 
> 

\ 


.886 


-10.421 




! 

.025 1 


.85 2 






( .019) 


( .543) 




_ . 


i 




CD 


1 

-1.213 


.745 


4 . 222 


- .487 


1 

.017 i 


1 

! .876 


After ^ 






( -071) 


(1.234) 


( . 066 ) 


J 


1 


WIPICS 


CES 


2.098 


.881 


-2.724 




.006 


.980 








( .016) 


( .293) 










CD 


2.934 


.884 


.674 


- . 179 


.020 


.893 


Across5 






(.017) 


( -035) 


(.023) 






WIPICS 


CES 


1.945, 


.907 


-3.424 




.031 


.778 








( .025) 


1-578]_ 









TABLE V.' Airframe Cost Functions 



400 observations from 0316 to 1350. 

^ 90 observations from 2090 to 2179. 

6 

Periods 1 and 2 above except dates ending in 3, 5, or 7 
deleted to reduce the number of observations to 343. This tech- 
nique was applied to the before and after case with essentially 
the same coefficients resulting as reported above. Thus, de- 
letion of some data should not have significantly affected the 
across WIPICS coefficients. 
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Function Coefficients SE R 




TABLE VI. Engine Cost Functions 



* Not significant at the 5% level. 



463 observations from 0184 to 1281 . 

8 

400 observations from 0215 to 1250. 

9 160 observations from 2158 to 2317. 
0 

623 observations for the CD Model. 
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All coefficients for the airframe functions are signi- 
ficantly different from zero. In addition to the two starred 
coefficients v^hich were not significant in the engine functions, 
the coefficient of In (^/L) in the first stage of the before 
WIPICS CES estimation was not significant. Since the 

functions were estimated from varying time periods the SE 

2 . • . . 

and R are representatxve of only the respectxve functxon and 

should not be compared across functions. Thus, to characterize 
the functions the Chow Test and statistical prediction inter- 
val discussed in Chapter III were used. 

B. STABILITY ACROSS THE CHANGE 

The Chow Test was used on the cost functions to determine 
if they were in fact different before and after V\/IPICS 
(Table VII) . 



Airframes 



Engines 



Critical Results 



Model 


Test Statistic 


Value 




CD 


16.94 


(4.362) 


Reject 






2.43 




CES 


23.98 


(4,362) 


Reject 






2.43 




CD 


8.30 


(4,615) 


Reject 






2.43 




CES 


(Not computed) 







TABLE VII. Chow Test Results 
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In all tests the hypothesis that the before and after WlPICS's 
cost functions are the same was soundly rejected. This im- 
plies the additional conclusion that a technological change 
‘'^may” have affected the relationship between imputs and out- 
puts , 

C. PREDICTION INTERVALS 

90% prediction intervals were calculated for the before 
and after WlPICS's functions by using the functions to pre- 
dict costs for the last 100 days (90 days for after WIPICS 
airframe functions) of this respective time periods. The 
same procedure used to construct Table IV was then used to 
construct Table VIII . 

The most striking result is the improvement obtained in 
percent of observations falling outside the prediction in- 
terval. This occurred because the 100 costs predicted were 
in fact a subset of the sample used to construct the cost 
functions. A second difference from Table IV is the size 
of the prediction intervals. The change was mixed, some 
larger and some smaller . This again points to the relatively 
unstable situation at NARFNI in which prices and amounts of 
inputs appear to be varying widely. 

Although the prediction intervals are non-symetric about 

\ 

the predicted cost, one half of the maximum interval should 
safely represent at least two standard errors of the cost 



39 



Model Interval Interval Interval 




d) 


C/5 




C/5 


u 


U 


U 


U 


0 


M 


0) 


M 


Mh 


CU 


4-> 


CU 


0) 


M 


Mh 


M 


DQ 




< 
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TABLE VIII. Extreme Widths of 90% 

Prediction Intervals (in dollars) 



function. This value will be used to make a significance 
statement about the cost comparison of operating with and 
without WIPICS. 

D. DIRECT COMPARISON OF PREDICTED COSTS 

Following the comparison procedures discussed in Section 

E, Chapter II, the cost differentials in Tables IX and X 
were obtained. 

CD CES 

Before WIPICS 
Data 

After WIPICS 
Data 



(BW-AW) (BW-AW ) 



$17885 


$3306 


$9241 


$7129 



TABLE IX. Difference Between Predicted 
Costs^^ - Airframes 



11 



Positive value indicates operation with WIPICS is 



estimated to be cheaper . 
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CD 


CES 




(AW- AW) 


(AW -AW) 


Before WIPICS 


$5126 


$5286 


Data 






After WIPICS 


$1152 


$3893 


Data 







TABLE X. Difference Between Predicted 

IP 

Costs - Engines 

Table IX is read as saying, "Based on the before and 
after WIPICS Cobb-Douglas cost functions it is estimated 
that WIPICS could have saved NARFNI $17,885 per day in the 
airframe program during the period 0316 to 1350, had WPICS 
been in operation at that time." Notice that this says nothing 
about changes which these cost functions cannot measure, 
such as an arbitrary change in the NORMS assigned to in 
coming repair items. These factors will be considered in 
Chapter V. In addition, the statement does not consider 
the inaccuracies of the models which were measured in Table 
VIII. It should be qualified by the assumption that the 
difference is not significant unless it exceeds, in absolute 
value, the sum of one-half of the two maximum prediction 



Positive value indicates operation with WIPICS is 
estimated to be cheaper . 
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intervals of the cost functions involved. In the above ex- 
ample this sum is $4776 (computed from Table VIII). Thus, 

statement is significant with at least 90% confidence 
(again, disregarding externalities to be discussed xn Chapter 
V) . 

Using the above procedure to interpret Tables IX and X 
the CD models indicate a significant advantage in the air- 
frame program and an insignificant disadvantage in the engine 
p];ogxa,m . The CES models indicate an insignificant WIPICS 
advantage in both programs . 

The above comparison could be refined by removing the 
effects of externalities. This should reduce the total 
variance of costs and it should reduce the maximum predic- 
tion intervals of the cost functions. The net result would 
then be a smaller region in which the cost differential would 
be considered insignificant, but also possibly smaller in- 
dicated cost savings. 
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V. VARIABLES EXOGENEOUS TO WIPICS 



The Cobb-Douglas and CES models do an imperfect job of 
estimating the effect of WIPICS on production costs due to 
many changes within the periods which were not caused by 
WIPICS. If the major changes were isolated, and the data 
corrected to account for them, then the proceedures discussed 
in this thesis should give an acceptable estimation of WIPICS' 
effect. Some of these changes are noted below. 

A. PRICE CHANGES 

Looking at the minimization problem of Chapter II , an 
increase in the price of an input would result in higher 
costs to produce a given level of output. Alternatively, 
a cost function estimated from data with low prices would 
tend to underestimate the cost of production for data taken 
when prices are high. For this reason increasing prices would 
depreciate the apparent effect of a technological change. The 
price of direct labor and penalty costs are accounted for in 
the cost functions used for this study so their fluctuations 
should not affect the final comparison of predicted costs . 

This is not the case for material prices and the prices of 
indirect labor absorbed into overhead. Shifts in these 
latter prices could, and shouild, be corrected by the appli- 
cation of a price index. 
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B. CDMMAND POLICY CHANGES 



This area is often elusive and difficult to identify with 
specific actions. However, obvious changes, such as a new 
ceiling on allowable overtime, should be accounted for. 

An increase in overtime, if not a result of command policy 
changes, may be caused by temporary inefficiency and shop 
conjestion due to the learning process that often follows a 
technological change. In this case the correction would 
consist of identifying the period affected and deleting it 
from the study , 

Another example is a shift in the standards for assign- 
ing NORMS at NARFNI . The shift is acceptable if experience 
shows that required labor hours have actually changed. Any 
other reason for the shift would result in an imaginary shift 
in the production function with a concurrent change in the 
derived cost function. This type of bias could also be 
corrected by application of an index to the data before 
estimating the cost functions. 

C. PERSONNEL TURNOVER 

A season of large personnel cuts followed by hiring of 
new personnel would naturally establish a training period 
in NARFNI . During this period the relative efficiency of 
workers would be low and a corresponding change in the cost 
function would be noted. If the training occurred during a 
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the function would 



period used to estimate a cost function 
inherit a bias resulting in comparison errors. The effect 
is complex, however, it may be that wages paid to personnel 
are low enough, relatively speaking, to allow the cost of 
production at a given level to remain approximately constant. 

-the relation between the inputs would be altered and 
a detailed study would be required to correct for the bias. 

D. CHANGES IN TYPE OF WORK REQUIRED 

In Chapter I it was explained that NARFNI has many different 
outputs in the form of repairs to various types of airframes 
and engines. These numerous outputs were aggregated into 
only two products, repairs of airframes and engines. The 
cost functions associated with these two products should do 
an acceptable job of predicting costs so long as the mix of 
work remains fairly constant within the respective programs. 

A shift in work type cannot be accounted for by the cost 
functions. Unfortunately, there is strong evidence that the 
work loads are shifting considerably. An approach to account- 
ing for these shifts is discussed in Section C of Chapter VI, 
PROBLEMS OF FURTHER RESEARCH . 
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VI. CONCLUSIONS AND PROBLEMS OF FURTIER RESEARCH 



A. EFFECTIVENESS OF WIPICS 

The comparison of estimated costs for the "with” and 
"without" WIPICS situations indicated a savings in the air- 
frame prograjTi and a loss in the engine program. The results 
of the two program comparisons are not additive due to the 
difference in time periods considered in each program. In 
addition, the component parts program was excluded from the 
study because of data format. For these reasons, an overall 
conclusion concerning the cost-effectiveness of WIPICS could 
not be extracted from this studj/ . 

Upon completion of the study it was found that WIPICS 
had not been fully operational in the engine and airframe 
programs for the periods originally stated. Numerous 
problems in implementing the system prevented a complete test 
of the system during this period. WIPICS has been completely 
operational in the airframe program since October of 1972 
but was not operational in the engine program as of 4 May, 1973. 

The indication of significant savings in the airframe 
program may be explained by shifts in the external variables 
discussed in Chapter V, a prime suspect being changes in the 
types of work. 
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B. AUDITING METHODOLOGY 



Investigation of the auditing methodology originally 
proposed by Spooner revealed eight primary steps that are 
essential to arriving at a strong conclusion: 

1. Identify external variations such as command policies 
and inflation of prices so they may be accounted for 
in the data. 

2. Correct the data for the variations identified in 
1 above so the situations before and after the 
technological change will be as similar as possible, 

3. Aggregate the data. 

4. Select the periods to study, keeping in mind that 
closeness to the change will aid in stabilizing 
functions while, at the same time, a delay may be 
needed to allow the technological change to become 
noticeable . 

5. Estimate the cost functions. 

6. Estimate the accurracy of the functions, 

7. Compare estimated costs derived from the cost functions 

8. Draw conclusions from 6 and 7 above. 

The audit conclusions may be more meaningful if identical 
time periods are used in each program studied (assuming there 
is more than one) . This will allow the separate effects to 
be additive for an overall statement about the effectiveness 
of the change. 
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Both the Cobb-Douglas and CES cost functions appeared to 
vary in ability to represent the production relation of inputs 
to outputs. Neither function dominated the other in accuracy 
therefore no conclusion can be made as to ivhich should be 
used. If only one is to be used the natural choice is the 
simpler Cobb-Douglas function. 

The results of the WIPICS test case indicate there remain 
problems with the accuracy of the tests. It is believed that 
relatively simple treatment of the data for external influ- 
ences would improve this accuracy. 

C. PROBLEMS OF FURTHER RESEARCH 

There are three primary areas in the auditing of cost- 
effectiveness analyses undergoing current research at the 
Naval Postgraduate School . 

First, Professor James Hartman is applying search tech- 
niques in an attempt to improve the estimation of the 
parameters in the CES cost functions. The 2-stage least- 
squares regression used in this thesis results in two estimates 
of the elasticity of substitution in the production function, 
both of which are significant . The problem is that the two 
estimates are invariably quite different. It is hoped that 
better estimates of this parameter will result in a more 
accurate CES cost function for use in the auditing methodology. 
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Second, Professor Norman K. Womer is investigating 
methods of accounting for the shifting of work type in the 
various programs which is believed to be responsible for much 
of the inaccuracies in the models used in this thesis . The 
principal effort in this area is toward developing a com- 
puter program to perform the regressions on raw data instead 
of the aggregated data. Dummy variables are being employed, 
one for each type of airframe or engine, to account for 
shifting work type. This requires a number of variables 
exceeding the capabilities of the computer soft -ware presently 
available . 

Finally, Captain Richard McGarrahan, USA, is investigating 
a second auditing methodology proposed by Spooner which 
employs a linear input-output production function instead 
of the continuous production functions assumed in this thesis. 
The objective of the work is to establish a basis by which to 
judge the abilities of the continuous models. The input- 
output model is expected to be more accurate in that it treats 
each type of work independently and should therefore be more 
stable. If the continuous models compare favorably in 
accuracy they would then be considered more desirable for 
their ease in handling. 
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